KETO-CTA Study

An Audit with the Actual Data

John | The John & Calvin Podcast

Dataset of Plaque Metrics

Longitudinal Data From the KETO-CTA Study: Plaque Predicts Plaque, ApoB Does Not


Plaque Data released by organization funding the study:

https://citizensciencefoundation.org/keto-cta/


All code, figures, data, studies available at:

https://github.com/SloughJE/keto_cta_analysis_check

Incorrect Figure 1B

Figure 1B from Study

Figure 1B from Data

Individual Change in Plaque Volume

(B) The red line represents the median change (0.8%), and the shaded area represents the IQR (0.3%-1.7%).

Incorrect Figure 1A

Figure 1A from Study

Figure 1A from Data

Individual Change in Plaque Volume

(A). The red line represents the median change (18.9 mm3), and the shaded area represents the IQR (9.3-47.0 mm3).

Incorrect Figures 2D, 2E, 2F

Figure 2F from Study

Figure 2F from Data

Changes in Total Plaque Score vs Coronary Artery Calcium

(C, F) Only CAC is associated with changes in NCPV and TPS. The regression line was fitted with the function “lm,” which regresses y~x, and the shaded area represents the standard error.

Nonstandard Percent Change

“The median change in NCPV was 18.9 mm3 (IQR: 9.3-47.0 mm³) and the median change in PAV was 0.8% (IQR: 0.3%-1.7%). Compared to baseline, these represent a 43% and 50% change, respectively.”

Standard percent change: \(\operatorname{median}\!\left(\frac{NCPV_{1y}-NCPV_{bl}}{NCPV_{bl}}\right)\times 100\%\)

“What was the typical relative change per subject?”


This is not what they did (on the Primary Outcome).

They computed a ratio of median change to baseline median: \(\frac{\operatorname{median}(NCPV_{1y}-NCPV_{bl})}{\operatorname{median}(NCPV_{bl})}\times 100\%\)

  • NCPV: \(18.9/44 \approx 43\%\); and PAV: \(0.8/1.6 \approx 50\%\)


Standard percent change vs. reported ratio-of-medians:

Outcome “ratio-of-medians” Median % change Mean % change
NCPV 43% 49.2% 81.4%
PAV 50% 47.3% 80.7%


Their “percent change” metric is nonstandard, not expected, and not explained.

Linear Model Assumptions

4 Simple Linear Regression Assumptions

3 are tested with data (in this situation)

  • Linearity: between the predictor and the outcome

  • Constant variance (homoscedasticity) of residuals

  • Normally distributed residuals


These linear assumptions are quantifiable and objectively testable.

  • If the assumptions don’t hold, statistical significance and uncertainty estimates aren’t trustworthy
  • Results may be invalid

LM Assumptions (claims vs tests)

“all linear model assumptions were corroborated with the R function performance::check_model.”

- Direct quote from the study

Actual Linear Model Assumption Tests

Model β Linearity Constant Variance Residual Normality
ΔNCPV ~ CACbl β = 0.18
p = <0.001		 Violation
p = 0.031		 Violation
p = 0.001		 Violation
p = <0.001
ΔNCPV ~ NCPVbl β = 0.25
p = <0.001		 OK
p = 0.198		 Violation
p = <0.001		 Violation
p = <0.001
ΔNCPV ~ PAVbl β = 5.48
p = <0.001		 Borderline
p = 0.050		 Violation
p = <0.001		 Violation
p = <0.001
ΔNCPV ~ TPSbl β = 7.37
p = <0.001		 OK
p = 0.132		 Violation
p = <0.001		 Violation
p = 0.001

Objective tests show all 4 models failed at least 2 tests each.

Letter to Editor and Response

“The validity of the regression models is also questionable. Despite claims of meeting assumptions, variables were reported using medians, suggesting non-normal distributions. Visual inspection of scatter plots shows clustering and no clear linear trends. Robust or nonparametric methods might have been more appropriate, and model diagnostics would improve transparency.”

- quote from letter to the editor

“we are aware of the relevance of linear assumptions to obtain accurate estimators. Because residual plot evaluation can also be subjective, we followed their suggestion and reran all models with robust linear regression (using the MASS::rlm function in R version 4.4.3 [R Foundation]). While, as expected, there were small differences with the published estimates, all models using robust regression were consistent with what was reported.”

- quote from the reply to a letter to the editor

“residual plot evaluation can also be subjective”

- quote from the reply to a letter to the editor

“we are aware of the relevance of linear assumptions to obtain accurate estimators. Because residual plot evaluation can also be subjective, we followed their suggestion and reran all models with robust linear regression (using the MASS::rlm function in R version 4.4.3 [R Foundation]). While, as expected, there were small differences with the published estimates, all models using robust regression were consistent with what was reported.”

- quote from the reply to a letter to the editor

LM Assumptions: Objective Tests

  • Calling residual plot evaluation “subjective” is misleading and evasive.

  • Visual checks are interpretive, but these assumptions objectively testable with quantifiable methods.

  • This is Statistics 101.


  • “reran all models with robust linear regression…MASS::rlm”
  • Robust Linear Regression mainly just down-weights outliers.
  • Does not deal with non-linearity, heteroskedasticity and non-normality of residuals.



check_model output:

Conclusions Without Supporting Models

“CONCLUSIONS In lean metabolically healthy people on KD, neither total exposure nor changes in baseline levels of ApoB and LDL-C were associated with changes in plaque.”

“there is no association between NCPV vs LDL-C or ApoB and TPS vs LDL-C or ApoB.”


Abstract claim component Model Model reported
Δ-plaque vs ΔLDL-C Δ-NCPV ~ ΔLDL-C Not reported
Δ-plaque vs LDL-C exposure Δ-NCPV ~ LDL-C exposure Not reported
Δ-plaque vs LDL-C baseline Δ-NCPV ~ LDL-C baseline Not reported
Δ-plaque vs ΔApoB Δ-NCPV ~ ΔApoB Reported
Δ-plaque vs ApoB exposure Δ-NCPV ~ ApoB exposure Not reported
Δ-plaque vs ApoB baseline Δ-NCPV ~ ΔApoB Reported
N/A NCPV_final ~ LDL-C exposure Reported (NCPV_final, PAV_final)

Missing TPS Models and Results

“Results Neither change in ApoB …, baseline ApoB, nor total LDL-C exposure … were associated with the change in noncalcified plaque volume (NCPV) or TPS.”

“Neither … change in ApoB nor the ApoB level … were associated … with TPS (Figures 2D and 2E, Table 3).” - “changes in and baseline levels of ApoB were not associated with changes in NCPV or TPS”


Figures 2D–2F are Δ-TPS (outcome) panels (vs ΔApoB, ApoB, CAC_bl)


Table 3 has no Δ-TPS models

No Δ-TPS ~ LDL-C, LDL-C exposure, or ApoB model results anywhere.

Bayes Factor rscale is not Moderate

“Bayes factors were calculated…with default settings and an ~ rscale value of 0.8 to contrast a moderately informative prior with a conservative distribution width (to allow for potential large effect sizes) due to the well-documented association between ApoB changes and coronary plaque changes.”

Calling rscale = 0.8 “moderately informative” is inaccurate.

R package docs: “medium”, “wide”, “ultrawide” = 0.354, 0.5, 0.707

  • rscaleCont = 0.8 is wider than “ultrawide” → a very diffuse prior that places substantial mass on very large effects
  • No reported prior-sensitivity to alternative r-scales
  • Same r-scale apparently used for all models, no justification.

Fixed description:

“We used a very wide prior on coefficients (rscale = 0.8, wider than the package’s”ultrawide”), which places substantial prior mass on very large effects. This diffuse prior penalizes small-to-moderate effects, requiring much stronger evidence than under the ‘wide’ or ‘medium’ defaults to support them.”

LDL-C Exposure Calculation

“LDL-C exposure on a KD was calculated by summing the products of the reported days on a KD prior to study commencement and baseline LDL-C on a KD plus the study follow-up days by their final LDL-C.”

\[ \text{LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \]


  • one baseline value for all pre-study Keto Diet time and one final value for all follow-up is a coarse simplification
  • relies on recall of KD start
  • standard AUC/time-weighted approach (need multiple measurements)
  • limitation due to resources

Lifetime LDL-C Formula Doesn’t Add Up

“LDL-C exposure on a KD was calculated by summing the products of the reported days on a KD prior to study commencement and baseline LDL-C on a KD plus the study follow-up days by their final LDL-C.”

\[ \text{LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \]

“Estimated lifelong LDL-C additionally included the product of age upon commencing a KD and pre-KD LDL-C.”

\[ \text{Life-LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \;+\; \boldsymbol{\big( Age_{\text{at-KD-start}}\cdot LDL_{\text{pre-KD}} \big)} \]

  • This equation is dimensionally inconsistent.
  • It adds days and age together. It’s like adding miles and inches.
  • pre-KD term is down-scaled by ~365× relative to the day-based terms.
  • any associations with outcomes are dominated by keto diet exposure.
  • if they did convert age, “lifetime” exposure just reflects how old someone was at KD

This isn’t an Age Mediation Analysis

“Estimated lifetime LDL-C exposure was only a significant predictor of final NCPV in the univariable analysis but lost significance when age was included as a covariate (Table 3). Both age and lifetime LDL-C exposure lost significance when baseline CAC was included in the model (Table 3).”

A mediation analysis is a specific statistical method that:

  • tests how an exposure affects an outcome through a middle step (the mediator)
  • estimates an indirect effect (through the mediator) and a direct effect (everything else).
  • requires a pre-specified pathway and proper statistical testing (usually with CIs or bootstraps).

This is not a mediation analysis.

  • No indirect effect was estimated or tested. No causal model specified.
  • Age can’t be a mediator of LDL exposure (age isn’t caused by LDL); it’s a confounder.
  • They only compared p-values after adding variables.

Age in Exposure, then Control for Age

“Estimated lifetime LDL-C exposure was only a significant predictor of final NCPV in the univariable analysis but lost significance when age was included as a covariate (Table 3). Both age and lifetime LDL-C exposure lost significance when baseline CAC was included in the model (Table 3).”

They ran (reported) three regressions with sequential covariate adjustment and concluded:

after adjusting for baseline CAC, neither age nor lifetime LDL-C predicts NCPV_final.

- CAC explains the association; age / lifetime LDL-exposure don’t matter.

“Loses significance” ≠ mediation. Collinearity since lifetime LDL-C embeds age.

Age is a common cause/proxy of both lifetime LDL-C exposure and NCPV (not caused by LDL), so it’s a confounder, not a mediator.


Adding baseline CAC (a downstream variable) blocks the mediation pathway and biases effects.


They did NOT report NCPV_final ~ lifetime LDL-C exposure.

Univariable Change Models Can Mislead

“Linear models on the primary (NCPV) and secondary outcomes were univariable”

All conclusions based on independent univariable change score models

Δ-NCPV ~ ApoB; Δ-NCPV ~ NCPV_baseline

  • Basic confounding unchecked (age, sex, BP)
  • Δ-NCPV contains baseline NCPV, biasing associations (mathematical coupling)
  • Δ results in less power, compounded noise
  • ignores starting plaque
  • baseline imbalance and shared correlates can inflate, reduce, or reverse associations

Consensus in biostatistics: use baseline-adjusted regression, ANCOVA style model: follow-up ~ baseline + covariates for pre/post analyses.

NCPV_follow-up ~ NCPV_baseline + ApoB_baseline + age + sex

Next-Year Plaque, Given Your Baseline

ΔNCPV ~ ApoB_baseline (unadjusted change score)

Question: Among all participants, is baseline ApoB associated with the raw change in plaque over 1 year, without accounting for where people started or other factors?

Slope interpretation: Average difference in change (mm³) per 1 mg/dL higher ApoB, unadjusted.

Clinically limited: Doesn’t condition on starting plaque, age, sex, BP; easily distorted by baseline differences and extra noise in change scores.
It doesn’t tell an individual with given baseline/risk profile what their 1-year plaque will be.


NCPV_follow-up ~ NCPV_baseline + ApoB_baseline + age + sex (baseline-adjusted regression)

Question: Among people with the same starting plaque, age, sex, and BP, do those with higher ApoB tend to have more plaque in 1 year?

Slope interpretation: Expected difference in 1-year NCPV (mm³) per 1 mg/dL higher ApoB, holding baseline NCPV and covariates fixed.

Clinically relevant: What a patient wants to know; “Given where I start and my risk factors, does higher ApoB mean more plaque next year, and by how much?”


Optionally test ApoB × NCPV_baseline to allow baseline-dependent progression.

Preprint vs Published: What Changed

Pre-print title: Plaque Begets Plaque, ApoB Does Not; Longitudinal Data From the KETO-CTA Trial

Published title: Longitudinal Data From the KETO-CTA Study; Plaque Predicts Plaque, ApoB Does Not

In text: find and replace ‘begets’ with ‘predicts’.


Description of main figure, and primary outcome:

“Most participants presented with stable NCPV (Figures 1A and 1B), with 1 participant exhibiting a decrease in NCPV (Figures 2A to 2C) and 6 participants showing decreases in TPS scores over 1 year”

Removed and replaced with:

“The median change in NCPV was 18.9 mm3 (IQR: 9.3-47.0 mm3) and the median change in PAV was 0.8% (IQR: 0.3%-1.7%). Compared to baseline, these represent a 43% and 50% change, respectively.”


In CONCLUSIONS section: “In an exploratory analysis, changes in and baseline levels of ApoB were not associated with changes in NCPV or TPS…”

“exploratory” added in published version. Only mention in entire paper.


Table 1 median (Q1–Q3) PAV at baseline changed from 1.25% (0.5–3.6) to 1.6% (0.5–4.9).

Record of pre-print is unavailable. All links go to new version. It is available here.

Conclusions

  1. Claims and visuals contradict the data
  1. Multiple headline conclusions lack supporting analyses/results
  1. Some metrics are nonstandard or conceptually unsound
  1. Core statistical assumptions are not met, despite the claim that they were “corroborated”
  1. Baseline-adjusted regression (ANCOVA) is the clinically relevant analysis

Miscellaneous issues

Location Study Text Issue
Figure 1 caption “…the shaded area represents the IQR (9.3-47.0 mm³)…” Imprecise. The quoted IQR is the ΔNCPV IQR; shaded band is 25th–75th percentile band at baseline and 1-year.
Figures 2D–F Caption “(D to F) Change in ApoB, baseline ApoB, and baseline CAC vs NCPV.” Axes/legend appear to be TPS, not NCPV → label/data mismatch
Table 3 caption “R² = squared correlation coefficient (explained variability).” Reports negative values, must be adjusted R²
Table 3 β = 0.18 No reporting of CIs for β.
KD Duration Table 1 vs Table 3 KD duration mean 1,642.7 days (≈4.5 y) vs caption “LDL-C exposure … mean 5.7 y.” Inconsistent durations.
KD Duration Abstract vs Table 1 LDL-C exposure median 1,302 d (984–1,754) vs KD duration median 1,427 d (1,002–1,938). Metrics conflated. exposure should be in units like mg/dL·days, not days.
Table 1 (Total cholesterol) Median 338, IQR (301–337). Impossible IQR (Q3 < median)
Table 1 Baseline Characteristics Baseline table provided. No follow-up characteristics table
multiple “changes in baseline levels” Baseline doesn’t change.
Software statement “using R 4.0.3 (2020)… with the last available version for Sept 2024.” Unclear what version was used.
Central illustration (quote) “Neither total exposure nor changes in baseline levels of ApoB and mg/dL were associated with changes in plaque. Conversely, baseline plaque but ApoB was not associated…” Incoherent/grammatical errors
Main text (Results/Discussion) “…baseline values magnify their percentual changes…” percent change or percentage-point change

Appendix: Plaque Metrics

Baseline, follow-up, and paired change in plaque metrics at 1 year
Metric N1
Median (Q1–Q3)
Mean ± SD
Baseline Follow-up Change2 Baseline Follow-up Change2
Noncalcified plaque volume (mm³) 100 44.0 (15.5–102.2) 66.0 (25.2–163.2) 18.8 (9.3–46.6) 75.9 ± 88.3 107.4 ± 112.7 31.5 ± 31.5
Coronary artery calcium score 100 0.0 (0.0–54.2) 0.0 (0.0–56.0) 0.0 (0.0–5.0) 50.3 ± 100.9 58.8 ± 120.4 8.5 ± 25.5
Percent atheroma volume (%) 100 1.6 (0.5–5.0) 2.6 (1.0–6.8) 0.8 (0.3–1.7) 3.2 ± 3.8 4.4 ± 4.6 1.2 ± 1.2
Total plaque score 100 0.0 (0.0–2.2) 1.0 (0.0–3.0) 0.0 (0.0–1.0) 1.7 ± 2.6 2.2 ± 3.0 0.5 ± 1.1
Calcified plaque volume (mm³) 100 0.3 (0.0–19.8) 0.9 (0.0–24.2) 0.5 (0.0–4.2) 18.7 ± 37.4 22.7 ± 42.5 4.0 ± 7.4
1 N = number of paired scans per metric.
2 Change = follow-up − baseline, summarized across participants.
Relative percent change from baseline to follow-up in plaque metrics at 1 year
Metric N (nonzero baseline)2
Percent change from baseline1
Median (Q1–Q3) Mean ± SD
Noncalcified plaque volume (mm³) 96 49.2 (26.0–104.4) 81.4 ± 85.5
Coronary artery calcium score 43 15.6 (1.8–45.7) 23.4 ± 56.6
Percent atheroma volume (%) 96 47.3 (25.9–100.0) 80.7 ± 86.3
Total plaque score 47 25.0 (0.0–50.0) 33.0 ± 77.4
Calcified plaque volume (mm³) 56 25.9 (10.8–64.7) 60.9 ± 104.2
1 Percent change = (follow-up − baseline) / baseline × 100; calculated per participant and summarized across participants.
2 N counts participants with a nonzero baseline for the metric.
Zeros, incidence (0→>0), and resolution (>0→0) by metric
Metric N Baseline = 0 Follow-up = 0 0 → >0 >0 → 0
Noncalcified plaque volume (mm³) 100 4 1 3 0
Coronary artery calcium score 100 57 54 6 3
Percent atheroma volume (%) 100 4 1 3 0
Total plaque score 100 53 46 10 3
Calcified plaque volume (mm³) 100 44 27 18 1
Percent change from baseline to follow-up: medians and trimmed mean
Metric N2 N baseline > 02
Median (Q1–Q3) % change1
Trimmed mean, 10%3
Overall (offset) Baseline > 0
Noncalcified plaque volume (mm³) 100 96 44.9 (23.7–77.8) 49.2 (26.0–104.4) 51.0
Coronary artery calcium score 100 43 0.0 (0.0–13.4) 15.6 (1.8–45.7) 5.1
Percent atheroma volume (%) 100 96 33.3 (19.9–56.3) 47.3 (25.9–100.0) 37.8
Total plaque score 100 47 0.0 (0.0–14.3) 25.0 (0.0–50.0) 5.0
Calcified plaque volume (mm³) 100 56 4.1 (0.0–18.1) 25.9 (10.8–64.7) 9.1
1 Overall % change = ((follow-up + c)/(baseline + c) - 1) × 100. BL>0 % change = ((follow-up - baseline)/baseline) × 100. c = 0.5 × the 5th percentile of non-zero values across visits for that metric, with floors: percentages 0.5; CAC 10; volumes 5. Positive values indicate increase vs baseline; negative indicate decrease.
2 N = paired participants; N baseline > 0 = number with nonzero baseline.
3 Trimmed mean removes 10% from each tail before averaging; computed on the Overall (offset) % change.

Plaque Metrics Charts

Plaque Baseline and Follow-up metrics are highly correlated


Follow-up NCPV was modeled as a linear function of baseline NCPV using generalized least squares (GLS): \(Y_i=\beta_0+\beta_1 X_i+\varepsilon_i\), with a power-of-the-mean residual variance \(\operatorname{Var}(\varepsilon_i)=\sigma^2\lvert\mu_i\rvert^{2\delta}\) (nlme varPower), where \(\mu_i=\mathbb{E}[Y_i\mid X_i]\).

Candidate variance structures (same fixed effects) were compared by likelihood-ratio tests under maximum likelihood.

The selected structure was refit by restricted maximum likelihood (REML) for parameter estimates and uncertainty.

Pointwise 95% confidence bands for \(\mathbb{E}[Y\mid X]\) were obtained from the model-based covariance of the fitted mean.

Fit statistics (REML): AIC = 874.15; BIC = 884.49; \(\log L=-433.08\)

Variance function: \(\hat\delta=0.526\) (95% CI 0.357–0.695)

Residual SD: \(2.097\) (df = 98; 95% CI 1.029–4.273)

Term Estimate 95% CI p-value
Intercept 8.547 5.504 to 11.590 <0.001
Baseline NCPV 1.304 1.231 to 1.378 <0.001

Robustness check (OLS with HC3 SEs): \(\hat\beta_1=1.251\) (SE 0.041), \(p<0.001\), consistent with the GLS slope

Diagnostics were acceptable: on the normalized GLS scale residuals were approximately homoscedastic and centered, with only mild tail departures; inferences were robust to tail behavior (HC3-robust SEs and pairs bootstrap CIs).

Plaque Baseline and Follow-up metrics are highly correlated


R² (baseline → follow-up, with linear mean): Baseline alone explains ≈ 96% of the variation (GLS weighted “R²” ≈ 93%) in follow-up values. Only ~4% (or 7%) is left after accounting for baseline.

ICC = Intraclass Correlation Coefficient — Proportion of total variance due to differences between people (vs within-person change).

ICC ≈ 0.91: about 91% of variability is between individuals; within-person year-to-year change is small.

This is Autocorrelation, more specifically test–retest reliability, not insight. Baseline plaque “predicts” follow-up mainly because the measure is relatively stable over time (high R², high ICC).

If we use baseline-adjusted regression, we could answer the question: “What explains the extra change beyond baseline?”

baseline-adjusted model (ANCOVA):

Follow-up ~ Baseline + ApoB (+ covariates)

– Test ApoB’s incremental value (partial F-test / ΔR²)

– still need to check model assumptions

Effect modification check:

Follow-up ~ Baseline + ApoB + (+ covariates) + Baseline*ApoB

– If the interaction is non-significant, ApoB’s association with follow-up does not depend on baseline level.